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ABSTRACT 


U.S. statutory policy requires the armed services to continuously balance 
manpower inventory with congressionally authorized requirements. Inaccurate 
forecasts put the Navy’s budget at risk and degrade overall mission readiness. 
Navy policymakers must be able to rely on accurate inventory forecasts to 
develop necessary manpower plans that steer inventory to match planned 
authorizations. Strength planners, in turn, rely on forecasting models like the 
Officer Strategic Analysis Model (OSAM) in an attempt to accurately predict 


future inventory levels. 


This study utilizes applications of data farming to OSAM to simulate 
Unrestricted Line Officer (URL) inventory over a seven-year period. Additionally, 
the research utilizes applications of Design of Experiments (DOE) to project 
Surface Warfare Officer (SWO) inventory across a variety of assumptions, 
including a proposed Enhanced Probationary Officer Continuation and Re- 
designation (EPOCR) policy. Analysis finds that current policy will reduce 
FY2016 URL inventory by 8% over a seven-year period, and over-execute SWO 
inventory authorizations by 40%. We find that EPOCR reduces operating 
strength deviation (OSD) in total SWO inventory strength by 12% by FY2022. 
Additionally, implementing a low accession plan and a high transfer plan is the 
most robust in correcting OSD. When implemented correctly, EROCR has the 
potential to decrease OSD to modest levels with minimal risk of under-execution. 
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EXECUTIVE SUMMARY 


U.S. statute requires the services to balance their manpower inventory to 
congressionally authorized requirements. As a result, Navy strength planners, 
community managers, and policymakers plan and manage the Navy’s end- 
strength, skill inventories, promotions, and accessions to meet the 
congressionally mandated end strength. To do so, they must continuously and 
accurately predict the Navy’s inventory strength by designator and paygrade 
throughout the fiscal year. Inaccurate prediction and resulting policies put the 
Navy’s budget at risk and degrade mission readiness. 


Strength planners rely on the accuracy of complex simulation models to 
correctly predict the Navy’s future inventory strength. Models help Navy 
leadership understand the potential impacts and risks inherent in current and 
planned policies to the force inventory structure. 


The Officer Strategic Analysis Model (OSAM) is an entity-based stochastic 
simulation model that allows for projection of annual future officer inventory and 
losses across communities by applying various historical trends and force- 
shaping policy to current inventory. The current version of the model, as used by 
strength planners, however, is stripped of its stochastic capability, which limits 
assessing risk in the model’s results. Additionally, OSAM, as currently used by 
strength planners, simulates only one scenario at a time, making the simulation 
model ineffective for the large-scale implementation necessary for investigating 
effects of manpower policies. 


This thesis uses concepts of design of experiments (DOE), data farming, 
and a stochastic enhancement of OSAM to forecast Unrestricted Line (URL) 
officer inventory. Data farming is used to simulate 100 replications of a base- 
case scenario that projects Navy Unrestricted Line (URL) officer inventory over a 
seven-year period. The base-case scenario assumes that current manpower 


policies and loss rates prevail over the projected period. We use the base case to 


xvii 


gain insight on the expected trend of URL inventory strength and the risks in 
future SWO inventory strength assuming the Navy continues with current 
manpower policy. We find that total operating inventory strength falls by an 
average of 8% over the seven-year period from 24,815 officers at the beginning 
of 2016, to 22,890 + 13 officers at the end of 2023. The mean end strength 
results in the base case have extremely small standard deviation values, 
suggesting little value in OSAM’s stochastic enhancement. More importantly, 
base-case results show SWO inventory strength will over-execute OPA by more 
than 40% in grade O3 over the entire projected period. 


Additionally, the study uses concepts of data farming, design of 
experiments, and High-Performance Computing (HPC) to simulate surface 
warfare inventory using key factors and assumptions, including a proposed 
enhanced Probationary Officer Continuation and Retention (EPOCR) policy. We 
use historical loss rate categories, accession plans, transfer plans, and three 
user-added force-shaping plans corresponding to EPOCR losses. The current 
year promotion plan is used and an auto-promotion method that constrains 
inventory to the Defense Officer Personnel Management Act (DOPMA) 
guidelines is assumed. Using resources available at the Naval Postgraduate’s 
Simulation Experiment and Efficient Design (SEED) Center, a Nearly Orthogonal 
Latin Hypercube Design for three discrete factors corresponding to EPOCR 
losses is crossed with a Full-Factorial Design for three categorical factors 
corresponding to the historical plans. A “landscape” of output is farmed, 
collected, and analyzed for insight. 


We use graphical and statistical analysis, meta-modeling, and robust 
analysis to gain insight from the results. We find that implementing EPOCR 
significantly reduces grade O3 over-execution by FY2022, and levels the 
average operating strength in all grades to match OPA. Total operating inventory 
strength is reduced by 12% over the seven-year period, from 8,365 officers at the 
beginning of FY2016 to an average of 7,375 + 739 officers at the end of FY2023. 
However, we also find EPOCR greatly elevates the risk of over-execution. To 


xviii 


mitigate this risk, we use robust analysis and metamodeling to find robust 
policies that control OSD. We find that implementing a low accession plan and a 
high transfer plan is the most robust in correcting OSD. Finally, we find when 
implemented correctly, EPOCR has the potential to decrease OSD to modest 
levels with minimal risk to OSD. 
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I. INTRODUCTION 


A. PROBLEM STATEMENT 


During the last decade, operating inventory, the available number of 
officers within the Navy’s Unrestricted Line (URL) Community, has continually 
exceeded its authorized billets (BA), particularly within its junior pay grades. This 
phenomenon, termed by Navy inventory strength planners as Officer Over- 
Execution (OOE), has imposed large financial costs on the Navy’s budget 
(Parcell, 2015). Comparisons of operating inventory to officer programmed 
authorizations (OPA), the authorized end strength, for four URL communities are 
depicted in Figure 1. Over-execution is particularly profound within the Navy’s 
Surface Warfare Community (SWO). The SWO community, like most URL 
communities, has a closed-loop personnel system. This, coupled with historically 
low retention at the mid-level grades, has traditionally led to more SWO 
accessions than is authorized. However, the large number of accessions, 
“apparently” needed to sustain the low retention rate, has resulted in manpower 
operating inventory overages in the junior ranks (Huff, et al., 2015). A paygrade 
profile for the SWO community comparing operating inventory to authorizations 
from 2012 to 2015 is shown in Figure 2. Grades O1 and O2 have high historical 
overages, but even more startling is the over-execution in grade O3, which 
exceed 40% in FY2014 and FY2015. If left uncorrected, over-execution is likely 
to persist. However, naively fixing over-execution by lowering accessions 
introduces risks to mission readiness as required billets may go unfilled. Any 
sound solutions require accurately projecting and predicting future operating 


inventory based on the inherently unpredictable future behavior of personnel. 


Inventory 


10000 - 





Inventory versus Authorizations (NFO, PILOT, SUB, SWO) 


NFO 





Desig | Inventory 
PILOT SUB SWwO || Authorized 








Figure 1. 
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As part of their long-range planning and programming tasks, officer 
strength planners at the Navy Strategic Resourcing Branch, N100, rely on 
powerful and complex manpower inventory models like the Officer Strategic 
Analysis Model (OSAM) to predict long-term trends in Officer Inventory. OSAM is 
an entity-based deterministic simulation model used to perform annual forecasts 
of officer inventory over the Future Years Defense Program (FYDP). Using 
OSAM for strength projection, however, is limited to single small-scale scenarios 
that provide limited insight to strength planners. Additionally, as a deterministic 
model with no variation in its results, OSAM does not provide any assessments 
on the risk associated with its results (Saluke, personal communication, April 26, 


2016). This thesis enhances OSAM’s utility by introducing randomness to the 


model and by using proven simulation techniques to run thousands of OSAM 


scenarios. 


B. BACKGROUND 


The Navy’s manpower costs have historically constituted a large 
proportion of the Department of the Navy’s budget and this proportion is 
expected to grow over the next fiscal years. The military personnel budget for 
active naval personnel (MPN) was $29.35 billion in FY 2016. This constituted the 
second-largest appropriation of a total Navy budget of $155.40 billion. This 
amount funded manpower requirements of 54,333 officers and 268,524 enlisted 
personnel for a total end strength of 322,857 (Department of the Navy, 2016). 


Title 10 of the United States code mandates that services regulate their 
inventory to authorized end strength, the number of uniformed personnel set by 
congress allowed on the final day of each fiscal year. The Defense Officer 
Personnel Management Act further places numeric constraints on the distribution 
of active-duty officers in grades O4 through O6 for any authorized officer end 
strength. Additionally, authorized end strength must be balanced to budgetary 
fiscal constraints. This responsibility requires a concerted effort among policy 
makers, Community managers, and strength planners to continuously plan, 
manage and monitor the Navy’s officer accessions, promotions, transfers, and 


skill inventories. 


Steering current operating inventory levels requires accurately projecting 
and predicting future inventory strength using current and future policy and other 
assumptions. The Office of the Chief of Naval Operations (OPNAV) Manpower, 
Personnel Training, & Education (MPT&E) Resource Management Division, 
Strategic Resourcing Branch, N100, is responsible for projecting current Navy 
officer inventory to end strength and developing, assessing, and recommending 
personnel strength management policies and force-shaping initiatives that will 


correct officer inventory strength to authorized end strength. Personnel policy 


derived from inaccurate projections may lead to over-execution, in turn affecting 


the Navy budget and mission readiness. 


C. NAVY STRENGTH MANAGEMENT 


Strength management concerns aligning operating inventory strength to 
authorized end-strength through manpower management policies encompassing 
accessions, promotions, transfers, and force-shaping initiatives. This section 
describes terms relating to Navy strength management and applicable to this 


study. 


1. Manpower versus Manning 


Manpower, also referred to as “spaces,” drives the need for inventory. 
Manpower encompasses unconstrained minimum quantitative and qualitative 
positions needed to fulfill all Navy missions and functions. Navy resource 
managers derive manpower requirements by translating national defense 
strategies and force structure policy decisions into manpower needs. In general, 
requirements are written to minimum skill, pay grade, and quantity for each unit 
or Navy platform. However, budget constraints further obligate policymakers to 
decide on which missions and associated requirements to fund. The process 
ensures that sufficient funds have been set aside to pay for “spaces” when 
personnel are assigned to a “space.” The number of authorized requirements is 
referred to as end strength. Specifically, end strength is the number of uniformed 
personnel set by congress allowed on the last day of each fiscal year, and is the 
target to which strength planners strive to match inventory. 


Manning, also referred to as “faces” or operating inventory, is the specific 
inventory of personnel at an activity, unit, or platform in terms of numbers, 
grades, and occupational groups or skills. Each year, the Chief of Naval 
Operations, through the Department of Defense (DOD), must submit proposed 
end-strength to congress for approval and funding. Fiscal constraints on 
authorized end strength and how well the Navy manages accessions, 


promotions, transfers, and to some extent, losses, directly affects the inventory 
that is allocated to funded requirements. 


2. Operating Strength Deviation 


Fiscal constraints, manpower management, and policy create differences 
between authorized end strength (spaces), and the operating inventory, the 
number of personnel available to fill authorizations (faces). The difference 
between the “spaces” and the “faces” is what this study refers to as operating 
strength deviation (OSD). Over-execution is deviation when operating inventory 
strength exceeds authorized end strength, the converse being under-execution. 
Strength planners have to be cognizant about operating strength deviation 
throughout the fiscal year. Deviation in end-strength exists in both enlisted and 
officer communities and is tracked by skill and grade. Once deviation has been 
identified, community managers implement strength management policies and 
force-shaping plans to guide future inventory levels, or strength, to authorized 
end strength. This thesis focuses on operating strength deviation within the 


officer Unrestricted Line (URL) community. 


3. Navy Strength Planning 


Navy strength planners are tasked with determining the effects that 
current and future accessions plans, promotion plans, and losses will have on the 
ability to balance personnel inventory to authorized end strength. Officer strength 
planners predict, plan and manage total gains and losses by skill and grade for 
each fiscal year to meet Officer Programmed Authorizations (OPA), the number 
of officers, by grade and designator, programmed to meet end strength. Strength 
planning is founded on predicting losses and gains, and generally follows the 


equation 


Strength, , = Strength,,,_,, — Losses + Gains ,4-1) (1.1) 


p(t-1) p(t-l) 


where p and t are subscripts representing pay grade and time, respectively. 


Strength planning is a proactive approach that can identify future operating 
strength deviations and provide justification for policies that ensure matching 
future inventory strength to authorized end strength. Strength planning is, 
however, a delicate and complex task requiring strength planners to rely on 
complex personnel inventory strength projection models such as OSAM. 


4. Inventory Strength Modeling 


The use of inventory strength projection models for military purposes 
dates back to the conception of military operations research (Abrams, 1957). 
Navy Strength Planners and Manpower analysts use these models, to predict 
how current manpower inventory levels evolve over time. The models help in 
assessing the risks associated with current and planned manpower policies. The 
Department of Defense uses various inventory strength models employing varied 
inventory projection methods and techniques. These models generally follow the 
projection structure of Equation 1.1, but they differ in purpose and design. The 
following sub-section discusses some of the key modeling methods with relation 
to OSAM. 


a. Aggregate versus Disaggregate Models 


DOD aggregate models project inventory as an aggregate of inventory 
across all specialty skills (Schank et al., 1997). Disaggregate models, such as 
OSAM, further partition projections by individual inventory attributes like 
occupation or skill, years of commissioned service, and pay grade, thus providing 
more detail than aggregate models (Schank et al., 1997). This study uses OSAM 
to project URL inventory strength by designator, pay grade, and years of 


commissioned service. 


b. Short-Term versus Long-term Models 


Short-term inventory strength models make monthly projections within a 
specified period in a given fiscal year. Long-term inventory strength models 
project inventory in annualized time steps, from the end of one fiscal year to 
another for periods longer than one year. OSAM is a long-term inventory strength 
model capable of producing annual inventory forecasts for periods beyond seven 
years (Schank et al., 1997). This study uses OSAM to project URL officer 
inventory from FY 2016 to FY 2023. 


Cc. Deterministic versus Stochastic Models 


A model is deterministic if the output of the model is wholly determined by 
the assumptions and input settings selected by the user. In a deterministic 
inventory model, each unique set of inputs will always produce the same 
inventory output, which makes it unnecessary to run the model more than once 
for a particular set of input values (Lucas, 2000). Stochastic models have 
randomness in their output results. OSAM uses pseudorandom processes in its 
modeling process to assign losses, promotions, accessions, and transfers, 
inherently making a stochastic model. Updates to the latest version of the model, 
however, removed the model's random characteristics, effectively making it 
deterministic (Mundy, 2014). Using OSAM deterministically, however, subjects 
results to the “flaw of averages,” where single number averages are fallaciously 
used to represent uncertainty (Savage, 2009). This study utilizes a modified 
version of OSAM that reinstates randomness. Chapter III of this thesis discusses 
this modification to OSAM. 


d. Static versus Dynamic Models 


A static inventory model does not depend on the time factor but rather 
describes the inventory at a specific set time and assumes an identical flow of 
inventory over time (Law & Kelton, 2000). Dynamic models such as OSAM 
project inventory from one period to another, making the time component integral 


to and explicit in the modeling process (Law & Kelton, 2000). This thesis uses 
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OSAM to study the evolvement of Unrestricted Line Officer inventory strength, 


including Surface Warfare Inventory, over a seven-year period. 


D. PROPOSED SOLUTIONS TO OVER-EXECUTION 


The prevalent over-execution within the unrestricted line community, and 
chiefly within the SWO community, has emerged in importance and focus to 
Navy leadership, particularly due to recent budget cuts. As such, Navy leadership 
tasked N100 for solutions. One such solution is lowering SWO accessions to a 
level that is just enough to cover authorizations within the junior ranks. However, 
naively fixing over-execution by lowering accessions introduces risks to mission 
readiness as future mid-level requirements may go unfilled, given the SWO 


community’s low retention rates. 


Another proposed solution is the Enhanced Probationary Officer 
Continuation and Re-designation (EPOCR) Board. EPOCR is a proposed force- 
shaping initiative that extends retention and continuation board eligibility to SWO 
officers with four, five, and six years of commissioned service (P. Saluke, 
personal communication, April 26, 2016). The Navy uses continuation and 
retention boards to separate from service or transfer to other communities 
officers who no longer have viable progression career paths within their parent 
communities. The SWO community typically reserves probationary force-shaping 
policy to officers with less than four years of commissioned service. As proposed, 
the policy will refer to the EROCR board up to 120 officers with four, five, or six 
years of commissioned service. This study investigates the impact of a low 
accession plan and the EPOCR policy on projected SWO inventory. 


E. PURPOSE AND OBJECTIVE 


The author’s efforts are focused on utilizing OSAM’s inventory projections 
capabilities to provide a quantitative approach to understanding the risk 
associated with different manpower policies to the future composition of URL 
inventory, including the Surface Warfare inventory. Specifically, the study 


projects FY 2016 URL inventory to 2023, assuming current manpower policy and 
9 


loss rates, and measures the deviation of operating inventory to planned 
authorizations by FY and designator. Additionally, the study assesses the 
implications of a low accession plan to SWO inventory. The EPOCR policy is 
also investigated to determine its impact on SWO future operating inventory. 
Stochastic enhancement to OSAM enables us to produce confidence intervals for 


projected inventory strength results. 


F; RESEARCH QUESTIONS 


The scope of this thesis is guided by the following research questions 
posed by N100. 


1. What is the expected short- and long-term trend in URL inventory 


strength assuming current policy and prevailing loss assumptions? 


OSAM’s dynamic application can produce inventory evolvement over a 
specified period. Stochastic enhancement of the model will allow construction of 


confidence intervals on inventory strength results. 


2. What are the long-term risks in inventory strength for the SWO 


community associated with current policy? 


OSAM’s disaggregate methodology allows us to focus analysis on an 
individual designator. Its long-term annualized projection allows us to track 
inventory strength trends for periods of great lengths. 


3. What is the impact of the Enhanced Probationary Officer 


Continuation and Re-Designation Board on SWO inventory strength? 


OSAM is flexible to the extent that one can model additional force-shaping 
plans using its force-shaping option. This thesis uses OSAM’s force-shaping 


option to model losses corresponding to officers referred to EROCR. 


The insight gained from answering these research questions will identify 
risks in current manpower plans and provide justification for adopting sound 
policies and plans that will guide operating strength to authorized end strength. 


Analyzing the effects of various accession plans will also provide an 
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understanding of the effect of reducing SWO accessions to future operating 
inventory strength. 


G. METHODOLOGY AND SCOPE 


This thesis combines data farming and the efficiency of design of 
experiments (DOE) with a stochastic version of OSAM to obtain analyzable 
data from the model. Applications to support data farming and the stochastic 
version of the model are developed and provided by the Naval Postgraduate 
School (NPS) Simulations, Experiments, and Efficient Designs (SEED) Center, 
a research center within the school’s Operations Department that focuses 
on enhancing DOD Research via_ simulation design and _ analysis 
(https://harvest.nps.edu). First, the OSAM input factors that contribute the largest 
variation in the output of interest are identified from the model, and with guidance 
from N100 manpower analysts. With operating strength as the response variable 
of interest, loss rate categories, transfer plans, and accession plans are chosen 
as the explanatory variables. We introduce three additional force-shaping factors 
for SWO inventory with four, five, and six years of commissioned service (YCS), 
respectively, as a way to control future SWO operating inventory strength. The 
latter represents a proposed policy that increases the number of junior SWO 
officers referred for lateral transfer or re-designation via the Probationary Officer 
Continuation and Re-designation Board (Borozny, 2015). The number of factors 
and their corresponding levels dictate the type of design used for the experiment 
since complexity is limited by available computing power (Kleijnen et al., 2005). A 
stochastic modification of OSAM is adapted to utilize multi-cluster computing and 
further scripts are developed to consolidate the model’s output to desired 
specification. The model’s output is analyzed using advanced statistical 
techniques incorporating regression tree analysis, robust analysis, and 
metamodeling to explore the relationship between OSAM’s input factors and 


projected inventory strength. 


H. BENEFITS OF RESEARCH 


OSAM’s utility as an inventory strength projection tool is largely untested, 
and its value largely underestimated. This research extends previous studies by 
Sibley (2012), Borozny (2015), and DeHollan (2015) by studying, validating, and 
improving upon OSAM'’s utility as an inventory projection tool. Insights gained will 
enable manpower analysts to effectively use the model to accurately project 
officer inventory strength. The methods used in this research can subsequently 
identify the true effects of OSAM’s inputs on end strength results, and measure 
risks associated with current and planned manpower policies to future inventory 
strength profiles. Additionally, the study will gain insight into the implications of 


the proposed EPOCR policy to future SWO inventory. 


I. LITERATURE REVIEW 


Human resources are the most valuable, yet more complex, and often 
most expensive asset of most organizations. Due to the uncertainty of human 
behavior, organizations often resort to simulation models to emulate their 
manpower systems and predict the short- and long-term composition of their 
future personnel inventory strength. A large amount of research has been 
conducted to support the development or improvement of manpower models for 
inventory strength projection and optimization within the context of the military. 
Sibley (2012) offers an extensive summary of such research. Whereas numerous 
manpower models exist within and outside of military application, there is little 
research that applies the concepts of data farming and design of experiments to 
manpower models. In fact, Antony (et al. 2012) finds that little attention has been 
given to the application of DOE outside the manufacturing context. The literature 
reviewed in this section focuses on some of the few studies applying data 
farming and DOE to military manpower models. 


Blosch and Antony (1999) apply design of experiments to the Rating 
Supply Chain Model (RSCM) to study risk in the United Kingdom’s Navy 
manpower planning. Specifically, their study uses DOE to identify key variables 
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that cause gapping, referring to a job that is not filled by a competent and 
qualified person, while on sea rotation. RSCM is a stochastic queuing inventory 
model that represents the flow of inventory through manpower positions and their 
interrelations with events such as recruitment, promotion, and losses. The model 
takes as input nine covariates and outputs distributions on queue lengths and 
waiting times within the queuing process. The study designed an experiment to 
evaluate the effects of three variables, sea-shore ratio, length of sea-draft, and 
billet ratios, and their interactions using one of Taguchi's orthogonal array 
designs. Results from the experiment allowed for analysis using analysis of 
variance (ANOVA) and other statistical techniques to determine the factors that 
influence gapping at sea. Although Blosch and Antony were conservative in the 
number of variables, their study revealed a promising application of DOE to 


manpower modeling within the military arena. 


Erdman (2010) applies the concepts of design of experiment to the U.S. 
Army’s Enlisted Specialty (ES) model. Erdman uses DOE to identify the variables 
with the greatest effect on minimizing the deviation between projected personnel 
inventory and authorized end strength. ES, like OSAM, is a dynamic inventory 
model that projects military inventory by grade and skill over a multi-year 
planning horizon. With 800,000 variables, ES adds an optimization application 
with 224,473 constraints to the inventory modeling process. Erdman evaluates 
52 objective function coefficients of the model that place the greatest weight on 
decision variables and he uses the Plackett-Burman (Plackett & Burman, 1946) 
approach to construct his experimental design (Erdman, 2010). The findings from 
his analysis for the Army’s G1 showed a possibility of a 14% reduction in 


deviation, or an average drop of 8,355 misalignments (Erdman 2010). 


Nelson (2010) uses DOE to evaluate the Navy’s Bureau of Personnel 
Metrics and Analytics Branch (BUPERS-34) Reenlistment model. The 
reenlistment model uses linear regression to forecast the expected reenlistment 
rate for the Navy’s prescribed three reenlistment zones, based on years of 
completed service, assuming current conditions. Nelson uses DOE to determine 
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the most influential variables in the model that predict the reenlistment rate for 
each zone. He uses a 3-factor, 2-level full factorial design for each zone and 
analyzes experimental results using multivariable regression and other statistical 
techniques. Nelson identifies a number of insignificant variables in the model and 
recommends alternative models that improve robustness and fit for each of the 
three enlistment zones (Nelson, 2010). 


The thesis research of Sibley (2012) provides the foundations on which 
this thesis is based, and is the earliest literature known to this author utilizing the 
concepts of data farming and DOE on OSAM. Sibley seeks to evaluate and 
improve the prediction capability of an earlier version of OSAM by examining the 
performance across the simulation model’s loss adjustment factors for the lateral 
transfer variable. With 90 factors of discrete levels, Sibley uses a randomized 
Latin Hypercube (LH) design and a Nearly Orthogonal Latin Hypercube design to 
vary lateral transfer rates and loss adjustment rates among the SWO and Human 
Resource (HR) officers for pay grades O1-O6. Data farming and DOE allowed for 
efficiently exploring a factor space that would otherwise have been 
computationally impossible to explore. Sibley’s work, the first application of data 
farming and DOE to OSAM, not only lays the groundwork for continuing work, but 
also provides proven analytical solutions to OSAM’s limitations adopted in the 


current version of the simulation model. 


DeHollan (2015) advances the work of Sibley (2010) by applying concepts 
of data farming and design of experiments to an updated version of OSAM, 
enhanced with randomness. DeHollan measures the stochastic variation in the 
enhanced model and determines the effects of an improving economy on the four 
largest designators of the URL community, specifically focusing on pay grades 
O3 through O6. His work differs from that of Sibley (2010) in that in addition to 
using data farming and DOE to validate OSAM’s behavior, he uses the model 
itself to analyze behavior and future composition of officer inventory based on a 
given real-world scenario. Since OSAM’s existing platform does not have the 
intrinsic capability to model economic factors, DeHollan simulates a three-year 
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period of an improving economy by introducing forced losses for three 
consecutive years. He uses a 257-design point NOLH design to vary 27 factors 
representing forced losses per pay grade per designator per year. He analyses 
the relationships among the factors from the results of his experiment using 
metamodeling and other statistical techniques. 


Borozny (2015), who did her thesis research on OSAM concurrently with 
that of DeHollan, uses data farming and DOE to project Navy Surface Warfare 
officer inventory over a six-year period. Using the same modified version of 
OSAM as DeHollan, Borozny studies the interactions among OSAM’s accession 
modeling methods; loss rates; and an aggregated force-shaping plan 
representing an EPOCR policy. Borozny finds that OSAM’s different loss rates 
cause little to no variation in the model’s end strength output, holding all else 
constant. Additionally, the loss rates resulted in counterintuitive results where 
higher loss rates increase end strength and vice versa. Borozny does not provide 
an analytical explanation for this surprising result, but this author suspects that 
varying accessions by accession methods, rather than by accession plans, over- 
compensates for the losses in end strength inventory induced by the loss rates. 
The current thesis research differs from Borozny (2015) by varying the accession 
plans available in OSAM rather than the accession methods. Additionally, this 
study models EPOCR force-shaping transfers at a disaggregated level, 


representing each of the three affected YCS with independent factors. 


The works of DeHollan and Borozny provide the literature that serves as 
proof of concept for augmenting OSAM’s inventory strength projection 
capabilities with data farming and DOE to answer questions relating to officer 


manpower policies, force-shaping, and personnel strength planning. 


J. NAVY STRENGTH PLANNING MODELS 


Navy strength planners and policy makers use a variety of strength 


planning models, some of which are discussed briefly in the following sections. 
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1. Navy Officer Personnel Planning System Model 


N100 currently utilizes the Navy Officer Personnel Planning System 
(NOPPS) model for its officer strength planning and projection purposes. The 
model is used in conjunction with accession and promotion models to develop 
officer inventory management plans that will guide inventory to authorized end 
strength. NOPPS uses historical loss rates and average inventory to predict 
monthly inventory across the Future Years Defense Program (FYDP). The 
forecasting utility of NOPPS is, however, inadequate and limiting, forcing strength 
planners to calculate forecasts outside of the model (Frank, 1993). 


2. Navy Enlisted Strength Planning Model 


The Navy Enlisted Strength Planning (NESP) model, also utilized by 
N100, is an event-based deterministic tool used to forecast the Navy’s enlisted 
end strength (DeSousa, 2015). NESP determines enlisted losses for a given year 
by applying historical loss rates to the number of enlisted sailors eligible to leave 
the Navy within that year. Much like NOPPS, discussed earlier, NESP’s 
forecasting ability is also limited in that strength planners need to calculate loss 
rates externally (DeSousa, 2015). 


3. Officer Strategic Analysis Model 


The Officer Strategic Analysis Model (OSAM), utilized for this study, is the 
primary inventory strength projection model used by Navy strength planners to 
project long-term annualized projections of active officer inventory strength by 
pay grade, skill, and fiscal year. The model is described in Chapter III of this 
thesis. 


This chapter gives an overview of the problem that is the focus of this 
research, inventory over-execution within the URL community, and describes it 
within the context of inventory strength planning. The chapter poses the specific 
research questions and the quantitative methods by which these questions are 


explored as well as the relevance of this research. The next chapter discusses 
OSAM, the inventory strength simulation model employed for this study. 


THIS PAGE INTENTIONALLY LEFT BLANK 


ll. OFFICER STRATEGIC ANALYSIS MODEL 


This chapter gives a basic overview of OSAM, including a general 
description, the model’s design, its development, and a general description of the 
model’s analytic approach and processing steps. DeHollan (2015) gives an in- 
depth description of the model’s development and architecture. 


A. MODEL OVERVIEW 


The Officer Strategic Analysis Model (OSAM) is a dynamic entity-based 
long-term simulation model used to project annualized personnel inventory 
strength by skill and grade over a user-specified time period. The model is one of 
many decision tools available to Navy manpower analysts and the primary model 
used by Navy strength planners to project long-term annualized active officer 
inventory strength. OSAM simulates inventory, dynamically, in yearly time-steps, 
by applying historical loss rates, accession plans, transfer plans, promotion 
plans, and user-defined forced losses to the beginning of year (BOY) inventory. 
The model applies current Defense Officer Management Act (DOPMA), DOD, 
and Secretary of the Navy (SECNAV) policies and constraints to its yearly 


inventory strength projections. 


B. DEVELOPMENT 


OSAM was developed in 2007 to fill a Navy void of decision tools capable 
of projecting active officer inventory. The model was developed by LMI, a 
government-consulting group, specifically for the Navy’s N14 (Manpower, 
Personnel, Training, and Education Catalog, 2015). As of the date of this writing, 
OSAM has not completed the Verification, Validation, and Accreditation (VV&A) 
process required for all simulation models intended for DOD use. Nevertheless, it 
has undergone extensive upgrades from its original version and has not only 
contributed to effectively supporting planning and budgeting of long-term officer 
inventory strength but also made it possible to convert the model into one that is 


data farmable. The executable application of OSAM was originally built in 
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Microsoft Visual Fox Pro, a defunct database language available only on 
government computers, limiting OSAM’s usefulness (DeHollan, 2015). Another 
limitation of the original version was that the model’s input was decentralized to 
multiple file locations, requiring manual adjustment of each file to adjust scenario 


settings, a tedious process that was prone to error (Sibley, 2012). 


The latest software version is built in Microsoft Visual Basic (VBA) and 
houses its data files in Microsoft Access. The transition to VBA and Access 
circumvents government restrictions, expanding OSAM’s access and use by 
Navy strength planners and manpower analysts. The current version also 
consolidates OSAM’s input data to a single database, removing the need to 
individually edit each file when adjusting scenarios and pushing the models 


toward conforming to data farming standards. 


C. SOFTWARE ARCHITECTURE 


OSAM follows the general structure of a typical simulation model. The 
model’s architecture comprises an input component, a simulation module, and 
output component. The input component is a set of permanent source tables built 
and stored in MS Access files. When a simulation is executed, the input section 
links to the module comprising a user interface and simulation algorithms. The 
interface captures and transfers user input settings and parameters to the 
simulation module that drives the simulation algorithms. Each scenario’s inputs 
and parameters are stored in text files outside the confines of the model, allowing 
for easily accessing and rerunning or modifying scenarios. After each simulation, 
the model saves output data as two database tables. The first table is a multi- 
year summary (MYS) report of forecasted inventory data pertaining to promotions 
selections, loss counts, transfers, and accessions. The second is a flow point 
table that includes detailed promotion data including the number of vacancies, 
promotions, and flow point estimates for each competitive category and control 
grade (04, O5, and O06). A scenario Analysis Tool built in Excel is included as 
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part of OSAM’s software. A representation of OSAM’s interconnecting systems is 
shown in Figure 3. 
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Figure 3. OSAM Database and Model Component Structure 
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D. SIMULATION MODELING OVERVIEW 


OSAM’s simulation process is similar in design and structure to Equation 
1.1 displayed in Chapter | of this thesis, expressing the relationship between 
beginning inventory, losses, accessions, and the end strength. The analytical 
process begins with a beginning of year (BOY) inventory that accounts for each 
active duty officer brought forward from the previous fiscal year (FY), and 
represented by pay grade, designator, year of commissioning, and date of rank. 
The model subtracts inventory from BOY based on forecasted losses and user 
specified losses, and applies accessions and promotion algorithms to determine 
the end of year (EOY) inventory. For a multi-year forecast, EOY inventory for the 
previous year is the BOY inventory for the subsequent year. Since OSAM is a 
dynamic long-term disaggregate model, this process is repeated for each year for 
each designator and pay grade, for all years defined by the user. Figure 4 shows 
an overview of OSAM’s analytical process. 
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Figure 4. Overview of OSAM’s Simulation Process. 
Source: Mundy (2014). 





E. OSAM MODELING PROCESSES 


This section discusses how OSAM models losses, promotions, accessions 


transfers, and accessions to forecast officer inventory 


1. Losses 


Forecasting losses represents the most complex aspect of personnel 
inventory strength projection and is the first analytical task of OSAM’s modeling 
process. Navy officer strength planners categorize a loss as either natural or 
forced, depending on whether the loss was deliberately contrived through a 
force-shaping policy. OSAM categorizes losses in a similar way as Natural losses 
and Force-out losses and adds an option to define user-added losses. 


a. Natural Losses 


Officer natural losses, strictly speaking, comprise losses other than those 
due to force-shaping policies. These include officers who discontinue their 
service through voluntary retirement, resignation, and discharge. The natural 
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losses in OSAM are an aggregate of natural losses, losses due to other than 
force-shaping policy and behavioral losses. OSAM derives natural losses using 
historical loss rates predetermined by the Navy Officer Personnel Planning 
System (NOPPS) Forecasting Model (NFM). NFM estimates loss rates by FY, 
loss types, designator, paygrade, and years of commissioned service (YCS) 
group. OSAM models natural losses by multiplying the historical loss rates 
derived from NFM by the beginning of year (BOY) inventory as denoted as 


LosseSp,,; (g,d) = LossRate(g,d)x InVgoy (g,4) (2.1) 
where LosseSprq (g,d) represents the projected number of losses in pay grade g 
for a specified designator d. LossRate represents the applied loss rate, and 


INVgoy is the inventory at the beginning of the projection year. 


b. Force-out Losses 


Forced or Force-out losses are driven by force-shaping plans which are 
policy decisions stipulated by law and Navy leadership to compel personnel to 
leave service. The primary force-shaping plan for Navy officers is higher-year 
tenure, which limits the number of years of commissioned service at any specific 
pay grade. Navy promotions occur within specified YCS groups, and the 
percentage of officers promoted within each group must fall within minimum and 
maximum percentages established by law (SECNAVINST, 2006). The flexibility 
to balance promotions with higher year tenure creates an effective force 
management tool for Navy policy makers and officer strength planners that can 
be used to shape the inventory composition of any officer community. OSAM 
accounts for forced losses by applying maximum YCS conditional constraints to 
each pay grade designator combination: 


[ g 24 AND YCS = YCS gerge (8.4) |OR| g <3 AND YCS 2 YCS yx (8.4) | (2.2) 


where, for each officer, YCS represents years of commissioned service group, 


YCSyax is the maximum allowable YCS group for grade g and designator d, and 
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YCSpetire represents the maximum allowable YCS group allowed prior to 
retirement (Mundy, 2014). 


c. User-Added Losses 


In addition to natural and forced-out losses, OSAM'’s flexibility allows a 
user to add additional loss constraints by either adding or reducing losses by pay 
grade, designator, YCS, or FY, using the force-shaping utility. The model adds 
user-added loss constraints by adding or decreasing the number of losses after 
force-out and natural losses have been applied to BOY (Mundy, 2014). One can 
specify user-added loss constraints by FY, pay grade, YCS group, designator, 
and community. For instance, selecting to increase losses by 20 applies 20 
additional annual losses in each FY of a specified year range, and distributes the 
losses within the specified grades, YCS, and designators specified. OSAM 
applies the losses and additions stochastically using distributions proportional to 
inventory at the BOY of the specified FY. This thesis uses OSAM’s force-shaping 
utility to enforce user-added losses representing SWO officers referred to the 
EPOCR board. 


2. Promotions 


The second step in OSAM’s modeling process is determining and setting 
promotions. OSAM models promotions in two phases, representing control-grade 


and fully-qualified promotions. 


a. Control Grade Promotions 


Control-grade promotions apply to grades O04 to O6. Promotions to these 
grades is subject to stipulations in DOPMA, which provides limits on the 
authorized end strength and the minimum time in grade or flow-points in each of 
these grades. DOPMA also establishes promotion zones. Zones are correlated 
with an officer’s time in grade and officers eligible for promotions are categorized 
as either “in zone” or “above zone,” depending on whether they have already 
been considered for promotion. DON guidelines further regulate promotion 
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percentage rates to these grades (SECNAVINST, 2006). All naval officers 
compete for promotions with other officers of the same competitive category, 
regardless of their designator. Surface Warfare, Submarine Warfare, Naval 
Aviator, and Naval Flight Officers, the designators that are the focus of this study, 
belong to the Unrestricted Line (URL) competitive category. The designators 
compete against one another for available promotions by grade at each selection 


cycle. 


OSAM provides four methods for modeling promotions. The first models 
promotions based on a predetermined promotion plan. The second sets 
promotions according to predicted vacancies. The third method promotes to the 
number of vacancies or until flow-points reach minimum thresholds. The fourth, 
“Auto-Promote,” promotes to the number of vacancies while remaining within 
DOPMA/DOD/SECNAV guidelines. This thesis uses the FY2016 promotion plan 
to project FY2016 and FY2017 BOY inventory, and “Auto-Promote” to model 


subsequent years. 


b. Fully Qualified Promotions 


Fully qualified promotions refer to officer promotions to the grade of O2 
and O3. Nearly all eligible officers will promote to these grades. OSAM 
automatically promotes officers to these grades without the need for user input. 


3. Transfers 


The third modeling process applies transfer rates to inventory. A transfer 
refers to the reassignment of an officer from one designator to another. Naval 
officers may transfer by way of upward mobility from a training designator to a 
primary designator, or laterally through a lateral transfer or re-designation board. 
OSAM applies upward mobility transfers from a training to parent designator 
automatically without user input. The model applies lateral transfers based on a 
user-specified transfer plan. Additionally, the model constrains transfers to only 
grades 01-06 and mimics historical transfer patterns by YCS and pay grade. 
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4. Accessions 


The last step of OSAM’s modeling process is applying accessions or gains 
to obtain the end of year (EOY) inventory. An accession refers to personnel entry 
into the Navy manpower system from the civilian sector. OSAM uses accession 
plans that are based on historical data of prior accession plans executed by the 
Navy. The plans specify the number of officer accessions by community and 
paygrade. Alternatively, users can choose either unconstrained or constrained 
accession methods to model accessions for each projected FY. The 
unconstrained method sets the number of accessions so that approved end 
strength is met or exceeded. The constrained method uses the same logic as the 
unconstrained, but restrains accessions so that total personnel inventory does 
not exceed approved end strength. This study uses an accession plan for each 
projected FY. 


F. USER INTERFACE AND SCENARIO SETTINGS 


OSAM’s graphical user interface (GUI), named the scenario editor and 
shown in Figure 5, provides a convenient way to build and run a simulation 
scenario. Each scenario is a set of input parameters and settings that include a 
user’s choices of preset input tables referred to as plans; methods for applying 
accessions, promotions, and transfers; and Structured Query Language (SQL) 
statements that are used as options to program any further deviations from the 
originally set parameters (Mundy, 2014). The GUI allows users to easily set input 
parameters, define the modeling options, and specify desired output for each 
scenario. The GUI’s simulation building process exposes the user to the various 
input parameters and modeling options via the Parameters, Promotions, 
Accessions, and Force-Shaping tabs, as shown in Figure 5. 
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OSAM V3.1.4.2 Parameters Promotions Accessions Force Shaping Advanced 
Adjust Forecast Horizon Help 
Scenario: 
new 1.ini First Projection Year 2016 v 
Open Last Projection Year 2022 v 
N 
oc Select Plans 
Save 
Select Accession Plan AccPlan_FY16 y 
Save As 
Change Output Database Select Promotion Plan PromoPlan_FY16_COPR v 
Senora Select OPA Plan OPA _Fall_2015 a 
D:\OSAM V3.1.4.2\OSAM 
V3.1.4.2\Families\Template Select Transfer Plan TransferPlan_FY16 % 
Family Folder\OSAM 
Output Database.accdb 
Select Loss Rates Plan LossRates_FY15 v 
Create New Plans SQL 
Save and Confirm 
Figure 5. Officer Strategic Analysis Model Graphical User Interface 


1. Input Parameter Settings and Modeling Methods 


The primary input parameters are specified via the Parameters and 
the Force-shaping page. On the Parameters page, users can specify the length 
of the forecasting horizon and select from a variety of input tables for 
accessions plans, promotion plans, Officer Programmed Accession (OPA) or end 
strength plans, transfer plans, and loss rates categories. The force-shaping page 
allows users to add or constrain losses in addition to the losses resulting from 
the loss rates. 


The Promotion page, shown in Figure 6, Accessions page, and Advanced 
page allow users to choose among the different modeling methods by which 
OSAM calculates or determines promotions and accessions, and transfers, 
respectively. On the promotion and accession page, users may elect to use pre- 
existing plans in the OSAM database or choose among the model’s modeling 
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methods to determine promotions and accessions, respectively. The advanced 


page allows users to specify outbound transfers by community and paygrade. 
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Create New Plans SQL 


Save and Confirm 





Figure 6. Officer Strategic Analysis Model Promotion Methods 


2. Simulation Model Output 


Once each simulation scenario is completed, OSAM consolidates and 
aggregates all output into multi-year summary (MYS) tables and flow point tables 
containing entity level information for every simulation year. MYS tables produce 
aggregated forecast data from which counts of beginning of year (BOY) 
inventory, end of year (EOY) inventory, losses, promotions, transfers, and 
accessions are generated. Flow point tables include detailed promotion data that 
includes the number of vacancies, promotions, and flow point estimates for each 


competitive category and control grades 04, O5, and O6 (Mundy, 2014). The 
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OSAM software package also includes an analysis tool, built in Microsoft Excel, 
which is used to analyze and track differences in scenario output. 


The basic software version of OSAM and the analysis tool, however, are 
not suited for large-scale experiments involving thousands of iterations as 
needed for this study. Data farming tools are built to enable running multiple 


OSAM scenarios without the need for user interaction. 


G. OSAM SIMULATION RUNTIME 


The runtime for a single iteration of an OSAM scenario depends on the 
length of the forecasted horizon and the computing power of the machine on 
which OSAM is housed. A one year forecast scenario takes approximately two 
minutes on a 64-bit windows computer with an Intel 2.4 GHZ processor. A seven- 
year forecast takes approximately eight minutes on the same machine. 
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lll. METHODOLOGY AND IMPLEMENTATION 


This chapter discusses the two primary methodologies used in this study, 
Design of Experiments and Data Farming. Design of Experiments (DOE) is used 
to generate and explore a broad range of OSAM scenarios. Data Farming 
substitutes OSAM’s GUI with computing scripts that automate the process of 
iteratively running and collecting data from OSAM scenarios. Additionally, the 
chapter discusses supplemental software tools used to transform OSAM into a 
data farmable model. 


A. DESIGN OF EXPERIMENTS 
1. Definition and Applications 


Design of Experiments (DOE) allows for systematically determining the 
relationships between a factor, an input affecting a process or system, and the 
response, the output of that process or system, using computational 
experimentation. Varying input factors singularly, as by haphazardly choosing 
one scenario at a time in OSAM, is time wasting, limits insight, and may produce 
inconclusive or even disastrous results (Kleijnen et al., 2005). With DOE, one can 
carefully and efficiently make controlled changes to the input factors in order to 
gain maximum insight on how the factors affect the outcome variable. The design 
is specified as a matrix where columns correspond to factors and rows to a 
unique combination of factor levels, referred to as design points (Kleijnen, et al., 
2005). The response surface generated from the designed experiment is then 
studied and analyzed through metamodeling to gain insight on which, and to 
what extent, the simulation model’s inputs affect the model’s output. 


2. Relevant Input Factors 


A fundamental initial step in the design process is identifying and selecting 
relevant input factors (Sanchez et al., 2014). A factor, in the context of 


simulation, is an explanatory variable to a process or system’s output that is 
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manipulated by the experimenter. Decision factors are variables that can be 
controlled in the real world by a decision maker as opposed to noise factors that 


happen randomly, and are therefore uncontrollable. 


The choice of factors to vary depends on the factors’ characteristics, 
available computing resources, and the intent of the underlying experiment 
(Kleijnen et al., 2005). This research seeks to use OSAM to study the impact of 
accession plans, and the EPOCR policy characterized forced losses of SWO 
officers with four, five, and six years of commissioned service. Loss rate 
categories, transfer plans, accession plans, and force-shaping plans serve as the 
primary input variables of interest. Table 1 lists a summary of the factors used in 
our experimental design and their associated levels. A discussion of the factor 


ranges follows. 


Table 1. Experimental Design Factors and Corresponding Levels 

















Variable 
Factor Type Levels 
Controllable Factors 
_-EPOCR 4 YCS Discrete [0, 120] 
~-EPOCR 5 YCS Discrete [0, 120 
EPOCR 6 YCS Discrete [0,120] 
Transfer Plan Categorical FY[09_10, 14, 15], No Transfer 
_ Accession plan _ Categorical ——_FY[12, 13, 14, 15] 
Uncontrollable Factors 
Loss Plans Categorical FY[06 ,07, 10, 13, 14, 15] 





3. Factor Range 


In order to build an experimental design, it is necessary to determine the 
ranges for the relevant input factors. This section lists and gives a general 
description and the range determination for all the decision factors and the 


uncontrollable factor used in this study. 
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a. Decision factors 
This section describes the decision factors used in this study. 
(1) Transfer plans 


URL communities depend on upward mobility transfers and to some 
extent lateral transfers to grow their non-accession inventory. Transfer plans are 
determined yearly and are effectively determined and controlled by strength 
planners and policymakers. OSAM programs inbound and outbound transfers for 
grades O1 to O2, dimensioned by pay grade, community, and FY of execution. 
Grades O1 to O6 transfer via training to parent designators or via probationary 
officer continuation retention (POCR) boards, whereas grades O3 to O6 laterally 
transfer into other communities. For this study, five transfer plans were chosen 
with guidance from N100 strength planners (P. Saluke, personal communication, 
April 26, 2016). 


(2) | Accession Plans 


Accessions are the primary method by which communities get new 
personnel inventory. One of the primary objectives of this study is to determine 
the risk in future inventory posed by a low accession plan. As with transfer plans, 
Navy policy makers determine and have control over what accession plans to 
implement, and therefore are categorized as control factors for the purpose of 
this study. OSAM programs accession using predetermined tables that represent 
historical accession plans dimensioned by designator and the FY in which they 
were implemented. For this study, we selected four accession plans by choosing 
the plans that correspond to the most recent plan, the highest accession rate, the 


lowest accession rate, and a moderate accession rate. 


(3) | EPOCR (Four, Five, Six YCS) 


EPOCR policy is simulated as user-added losses. OSAM’s flexibility 
allows users to increase or decrease losses by community, pay grade, FY and 


YCS after the natural losses are applied via its force-shaping functionality. This 
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thesis uses OSAM’s force-shaping functionality to model losses representing 
SWO officers referred to the EPOCR board. Each YCS group is represented as 
an independent factor corresponding to SWO officers with four, five, and six 
years of commissioned service. We model EPOCR losses with a range of 0 to 
120 losses for each YCS category. This range represents the proposed total 
number of officers referred to the EROCR board. 


b. Noise Factors 
This section describes the uncontrollable factors used in this study 
(1) Natural Loss Rates 


Natural losses represent the uncontrollable factors in manpower modeling 
caused by inherent uncertainty in human behavior. For this reason, they are 
often the most complex factors to model. OSAM uses tabulated historical loss 
rates dimensioned by pay grade, community, and the FY or period in which the 
loss happens. (See Chapter .l) This study seeks to determine the variability in 
OSAM’s personnel end strength inventory projection over the uncontrolled 
losses. The study uses seven loss rate categories that represent a varied 
continuum of years with minimal losses to years with severe losses over the past 


ten years. 


4. Choice of Experimental Design 


Many competing design exist in the field of simulation analysis. The choice 
for which design to use, however, is significantly influenced by the number of 
factors, type of factors, and available computing resources (Kleijnen at al., 2005, 
Sanchez & Wan, 2012). This study uses a combination of three multi-level 
categorical and three discrete factors with a wide range of values. The relatively 
small number of factors, both categorical and, calls for a mixture of a gridded 
design and an efficient space-filling design applied to the categorical and discrete 
variables, respectively. Such a crossed design ensures all combinations of 
categorical factors are captured and reveals interactions and other possible 
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nonlinearities within the results. A full factorial design is appropriate for the 
categorical factors as it will examine the categorical factors at all levels and 
guarantee that all main effects and interactions are captured (Sanchez & Wan, 
2012). Nearly Orthogonal Latin Hypercubes (NOLH) arise as the most suitable 
design for the discrete variables given the broad range of values for the SWO 
forced losses. These designs, developed at the Naval Postgraduate School, offer 
excellent space-filling properties with minimal sampling (See Sanchez et al., 
2012, and Cioppa & Lucas, 2007). They also enable analysts to fit a broad range 
of meta-models and generate a diverse set of graphs. For additional families of 
NOLHs, see Hernandez et al., 2012 and MacCalman et al., 2016. 


5. Experimental Design Construct 


A full factorial design for the categorical variables is generated using the 
Full Factorial Design tool in JMP Pro 12.0.1 (JMP, 2015). The three categorical 
factors with six, four, and four levels, respectively, are crossed to get a design 
matrix of 96 design points. Figure 7 depicts the first 22 of the 96 design points for 


the full factorial design. 
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nes d ; 22 LossRates_FVO7 TransferPian FY14 AccPlen_FY'13 
@ Liv 
Figure 7. Full Factorial Design for the Three Multi-level Categorical 


Variables Generated Using JMP Pro 12.0.1 


An NOLH design is generated using Excel spreadsheet design template, 
created by Professor Susan Sanchez at NPS and available at the SEED center's 
website https:/harvest.nps.edu. The particular design template used generates 
33 design points and can accommodate up to 11 factors, more than sufficient for 
our three discrete factors. Figure 8 shows the complete design matrix with the 33 
design points, where, for instance, YCS_4 is a variable representing SWO 
officers with four years of commissioned service who have been referred to the 
EPOCR board. 
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low level 1 1 1 
high level 120 120 120 
decimals 0 0 0 
factor name YCcsS_4 YCS_5 YCS_6 





Figure 8. NOLH Design—3 Factors and 33 Design Points 


The design point matrix for the categorical factors is then crossed with the 
design point matrix for the discrete factors using JMP. The crossing generates a 
mixed design matrix with 3,168 design points. Figure 9 depicts a scatter plot 
matrix revealing the space filling properties of the resulting crossed design. Each 
subplot within the scatter plot matrix shows a two dimensional projection of the 
entire design and the levels used for each factor. 
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Scatterplot Matrix for Crossed Design 
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Figure 9. Scatter Plot Matrix for the Full Factorial Categorical Design 


Crossed with the Discrete Variable NOLH Design 


6. Replication 


Replication is the repeating of the entire design matrix in a simulation 
experiment. Replication is necessary to gain precision in its output results and to 
estimate the experimental error in the design if using a stochastic model 
(Cavazzuti, 2013). However, a tradeoff between the number of replicates per 
design point and the number of total design points is considered when 
constructing a design (Kleijnen et al., 2005). Previous studies using the 
stochastically enhanced version of OSAM have shown the model to depict very 
little random variation in end strength results. Borozny (2015) used 100 
replications for her experimental design and found little stochastic variation in 
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OSAM’s end strength results. DeHollan (2015) found similar results using 100 
replications for a single base case design point. This thesis considers these 
findings, as well as conclusions from a base case scenario simulated for this 
study, in deciding a tradeoff between replications and the total number of design 
points to simulate. We use five replications for each of the 3,168 design points for 
a total of 15,840 simulation runs. 


B. DATA FARMING 


Data farming, in technical terms, is the process of using rapid prototyping, 
simulation modeling, experimental design, high-performance computing, and 
data analysis to study questions of interest with large possibility spaces (Horne et 
al., 2014). The process is analogous to agricultural farming. Where farmers 
manipulate the environment and cultivate land to maximize agricultural yield— 
pest control, irrigation, fertilizer, etc —data farmers manipulate simulation models 
with designed experimentation to produce a landscape of output. A model is data 
farmable if it is possible to programmatically modify its input and automatically 
start an instance of that model, usually with a computer’s command-line interface 
(Horne et al., 2014). 


OSAM in its original version is not configured for data farming as the 
model is incapable of running multiple scenarios or replications of single 
scenarios successively without user interaction. The external software is 
necessary to data farm OSAM. Steve Upton, a Faculty Associate for research at 
NPS’s SEED Center developed OSAMFarmer, a supplemental data farming 
wrapper that is a collection of two key software programs, OSAMRunner and 
SimpleFarmer. Together, these programs “wrap” around OSAM to make the 


simulation model data farmable. 


1. OSAMRunner 


OSAMRunner strips out the graphical user interface and allows a user to 
run successively one or more replications of an OSAM scenario by feeding input 


specifications to OSAM via a command line interface. The program takes as 
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input; the model file; a configuration file specifying OSAM input parameters and 
settings for a specific scenario; and a seed argument, where the number of 
seeds corresponds to the number of desired replications. The program 
automatically saves the model’s output from each completed replication and 
successively starts the next simulation until all replications are exhausted. A user 


can then consolidate and analyze the model’s output. 


2. SimpleFarmer 


The second of the two programs, SimpleFarmer, concurrently schedules 
and manages a set of predefined tasks on a single, multi-processor machine by 
distributing tasks across individual processors. The program takes as input two 
task files, one that labels the names and locations of OSAM input task elements 
including the experimental design, and another containing a set of tasks 
corresponding to the simulated design points. SimpleFarmer schedules and 
executes tasks based on predefined dependencies and the number of available 
processers. Since SimpleFarmer distributes OSAM runs across processors, one 
can use the software to take advantage of parallelized processing on high- 


performance computer cluster nodes to cut down on the model’s run-time. 


3. Base Case 


In addition to the experimental design, this study includes a base case 
scenario to serve as a baseline for comparison of the experimental results. The 
base case contains the same input factors and parameter settings used for the 
experimental design and uses the most recent and prevailing manpower policies 
as represented in OSAM. No force-shaping policies are specified in the base 
case scenario. Input parameters include the FY 2016 accession plan, FY 2016 
transfer plan, and FY 2015 loss rates. The promotion method is set to FY 2017 
promotion plan for FY 2016 and 2017, and auto-promote for successive years. 
We replicate the base case scenario 100 times to further explore the stochastic 
variation in end strength results, and employ OSAMRunner to data farm OSAM. 
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C. EXPERIMENTAL RUNTIME 


The runtime for a single iteration of an OSAM scenario depends on the 
length of the forecasted horizon and the computing power of the machine on 
which OSAM is housed and run. Consequently, for this study, the total 
experimental runtime is the total time it takes to run all 15,840 of the simulation 
runs specified in the experimental design, each projected to seven years. A 
seven-year projection of OSAM takes approximately eight minutes to run using a 
64-bit windows computer with an Intel 2.4 GHZ processor. Without the benefits of 
parallelized cluster computing, this experiment would take approximately 88 
days. With the help of NPS’s SEED center resources, this study takes advantage 
of OSAMFarmer’s task distribution and scheduling properties to utilize a High- 
Performance Computing (HPC) multi-processor cluster system housed at NPS’s 
SEED center. The HPC reduces the total experimental time to nine days (S. 
Upton, personal communication, July 14, 2016). 


This chapter explains and gives justification for the methodologies chosen 
to explore the research questions posed in Chapter |. This chapter specifically 
describes the approaches taken to generate simulated data from OSAM. The 
next chapter presents and analyzes the results. 
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IV. ANALYSIS OF RESULTS 


This chapter presents and discusses the output from the base case 
scenario and the experimental design using graphical and statistical tools. We 
analyze the base case output to determine the degree of stochastic variation in 
projected operating strength. Additionally, we use results from the base case to 
determine the expected trend in URL inventory strength assuming current policy 
and most recent loss rates. We then analyze experiment data to gain insight on 
the effect of accession plans, transfer plans, and loss rate categories on 
operating inventory strength. We further study the impact of EPOCR policy. 
Using robust design, regression tree analysis, and meta-modeling, we determine 
the policy’s implications on operating strength deviation (OSD). 


A. ANALYTICAL TOOLS 


We use R version 3.3.0 (R Core Team, 2016) to consolidate, filter, and 
organize the results from the base-case simulation. R is also used for initial 
exploratory analysis of the base case results. We use JMP Pro Version 12.0.1 
(JMP Pro 2015) to process, organize, and filter the raw experimental output data 
into analyzable form as well as for initial exploratory analysis. JMP’s analytical 
and graphical tools are used to further explore the base case and experimental 
data for insight. 


B. ANALYSIS OF BASE-CASE RESULTS 


Output from the base-case scenario is used to assess the stochastic 
variation induced by the modification of the latest version of OSAM used for this 
study. Additionally, the base case gives a projection of the expected operating 
inventory strength using standing policy and assuming the most recent applicable 
loss rate category. This helps to determine the expected direction of current 
operating inventory assuming the assumptions prevail over the projected period. 
Specifically, the base case simulates 100 replications of a single design point 


with FY2016 accession plan, FY2016 transfer plan, FY2015 loss Rate, and 
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FY2017 promotion plan for FY 2016 and 2017, and auto-promotion method set 


for successive years. 


1. Data Processing and Collection 


OSAMRunner saves the output from each of the 100 design points as a 
separate CSV file. Each file contains multi-year summaries, for all 70 
designators, from which counts of beginning of year (BOY) inventory, end of year 
(EOY) inventory, losses, promotions, transfers, and accessions by designator 
and pay grade are generated. We use R to consolidate the CSV files and filter 
the multi-year summaries to inventory strength counts of the specific URL 
designators of interest, SWO, SUB, PILOT, NFO, and their corresponding 
training designators. We further group the data by design point, FY, Grade, and 
Designator. Training designators are summed together with their corresponding 
primary designators. For instance, SWO trainee is grouped and summed with 
SWO. 


2. Stochastic Variation 


To determine stochastic variation in the base case, counts of inventory 
strength are summed over all designators, grouped by FY, and averaged across 
the 100 replications. The results show that OSAM produces very little variation in 
URL projected inventory strength results across the 100 replications of the base 
case scenario. Figures 10 and 11 depict the variation in the URL and SWO 
inventory strength, respectively, projected across the 100 replications over a 
seven-year period. The lines depicted in the figures correspond to different 
replications. Variance in the total projected inventory increases in later years in 
both cases although the variance is small relative to the total inventory. 
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Figure 10. URL Total Inventory Strength Projected across 100 
Replications of Base Case Scenario 
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Figure 11. SWO Inventory Strength Projected across 100 Replications 
of Base Case Scenario 
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Summary statistics for URL inventory for each projected year are shown in 
Table 2. The standard deviation measures the variability of the projected values 
from the mean. FY 2023 has the highest variation in projected strength with a 
standard deviation of 12.49 and a mean of 22,890 officers, suggesting that 
stochastic enhancement of OSAM accounts for less than 0.06% of the variation 
in projected inventory strength. Similar results for SWO projections are shown in 
Table 3. 


Table 2. Summary Statistics of URL Projected Inventory Strength across 100 
Replications of Base Case Scenario 


Std. 
| DY =\V 


Pde) 10) 75th 
Quantile Quantile 


FY Mean 


Strength m7 wes 








2016 (BOY) 24815 . : : . . 

2016 24958 2.75 24952 24965 24956 24960 
2017 24545 6.28 24521 24556 24541 24549 
2018 24204 8.57 24176 24222 24199 24210 
2019 23911 10.67 23884 23936 23903 23918 
2020 23587 11.27 23558 23618 23579 23594 
2021 23304 12.42 23271 23334 23295 23312 
2022 23073 12.72 23023 23101 23066 23081 
2023 22890 12.49 22826 22909 22885 22899 





Table 3. Summary Statistics of SWO Projected Inventory Strength across 100 
Replications of Base Case Scenario 


Mean A , 25th 75th 
a See a mp UES Quantile Quantile 
2016(BOY) 8365 - - - - - 
2016 8405 1.8 8402 8409 8404 8407 
2017 8302 3.84 8294 8312 8300 8305 
2018 8229 5.1 8215 8240 8225 8232 
2019 8161 5.89 8148 8178 8156 8165 
2020 8066 7.03 8052 8085 8061 8072 
2021 7993 8.29 7967 8010 7987 7999 
2022 7950 10.3. 7920 7976 7943 7957 
2023 7910 12.41. 7878 7939 7902 7918 
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3. SWO Operating Strength Deviation-Base Case 


One objective of this study is to determine the risk in operating strength 
deviation presented by current policy. Since the base case parameters are set to 
reflect current manpower management policies, we use SWO inventory strength 
output from the base case to determine inventory risk by comparing projected 
operating strength to planned Officer Program Authorizations (OPA). SWO OPA 
given by grade and FY and programmed to FY 2022 are given in Table 4. 


The mean of the projected operating inventory strength by grade and FY 
compared to planned authorizations is shown in Figure 12. The total inventory 
strength composition for the SWO community is largely unchanged for FY2017 
and beyond. Standing out is grade O83 inventory, which over-executes 
authorizations by more than 40% in all projected FYs. There is significant 
increase in FY2017 OPA for grade O1, causing significant under-execution in this 
grade that persists to the end of the seven-year projection period. 


Table 4. SWO Officer Programmed Authorization by Grade and FY 
Source: N100 (2016) 


ad ENS LTJG LT LCDR CDR CAPT 
2016 1528 1091 1388 1096 657 256 
2017 2289 1371 1570 1082 651 243 


2018 2152 1379 1596 1096 654 241 


2019 2104 1442 1608 1111 660 241 | 


2020 1993 1442 1623 1103 661 238 


2021 1988 1441 1624 1101 661 238 


2022 1988 1441 1624 1101 661 238 
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Figure 12. Projected SWO Inventory Strength against Officer 
Programmed Authorization by Grade and FY (Base Case) 


C. INITIAL ANALYSIS OF EXPERIMENTAL DESIGN DATA 


This section describes and assesses the experimental design data, with 
EPOCR policies, generated by the constructs of data farming and design of 
experiments described in the previous chapter. Whereas the experimental data is 
used, in the context of this research, to assess the implications of manpower 
policy plans and EPOCR policies on future Surface Warfare inventory strength, 
the initial analysis assists in finding insightful information and extracting 
elementary conclusions from the experimental data itself. We describe the 
collections and processing of the experimental design data and analyze the 
effects of the key input factors on operating inventory strength. Regression tree 
analysis is used to determine the significant contributors to FY2022 total 


operating inventory strength, for all replications in the experimental design. 
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ie Data Processing and Collection 


Experimental data is consolidated and saved as Comma-Separated 
Values (CSV) file. The output file contains results for all the 15,840 design points, 
with each row in the CSV file corresponding to a design point. The data is 
structured as a multi-year summary report containing all designators and from 
which counts of beginning of year (BOY) inventory, end of year (EOY) inventory, 
losses, promotions, transfers, and accessions by designator and pay grade are 
generated. The file also includes a column for the random seed used for each 
design point, and columns representing each of the six factors in the 
experimental design. A format of the raw dataset produced from the experimental 
design is shown in Table 5. Since the raw dataset is all-inclusive, containing 
strength and loss counts for all 78 officer designators, and paygrades, including 
flag officers represented as grade O7, we use JMP to subset the raw data to 
retain only SWO and SWO trainee inventory counts for grades O1-O6, and 
further filter the count status to inventory strength rather than loss counts. The 
resulting dataset is treated as a parent dataset from which subsequent datasets 
are constructed for further analysis. The format of the resulting parent dataset 
showing the first 10 observations is shown in Table 6. 


Table 5. Raw Data Format of Experimental Design Results 


EPOCR EPOCR EPOCR 


DP LOSS RATE TRANS PLAN ACC PLAN (YCS4) (YCS5) (Y¥CS6) seed FY Grade Skill Status EOY Totals 
1, 1 FY10 FY09 11 FY14 120 11 53 25841483 2016 1 111x - Surface Warfare 1 Loss - Natural 2 
2. 1 FY10 FY09_11 FY14 120 11 53 25841483 2016 1 111x - Surface Warfare 4 Loss - Force-Out 1 
3. 1 FY10 FY09_11 FY14 120 11 53 25841483 2016 1 111x - Surface Warfare 5 Loss - User Ad... 2 
4 1 FY10 FY09_11 FY14 120 1 53 25841483 2016 1 111x - Surface Warfare 8 No Change 167 
5) 1 FY10 FYO9_11 FY14 120 11 53 25841483 2016 1 112x- Submarine Warfare 8 No Change 35 
6 1 FY10 FY09 11 FY14 120 11 53 25841483 2016 1 113x - Special Warfare 8 No Change 57 
7, 1 :FY10 FY09_11 FY14 120 1 53 25841483 2016 1 114x - Special Operations 8 No Change t 
8 1 FY10 FY09_11 FY14 120 11 53 25841483 2016 1 116x - Surface Warfare Trainee 1 Loss - Natural 33 
9 1 FYI0 FYO9_11 FY14 120 " 53 25841483 2016 1 116x - Surface Warfare Trainee 7 Accession 795 
10, 1 FY10 FYO9_11 FY14 120 1 53 25841483 2016 1 116x - Surface Warfare Trainee 8 No Change 683 
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Table 6. Data Format of the Filtered “Parent” Dataset Derived from the 
Experimental Design 


4 


EPOCR EPOCR EPOCR N 
LOSS_RATE TRANS_PLAN ACC PLAN (YCS 4) (YCS5) (YCS 6) seed FY Grade Rows EOY_Totals 


DP 
1 1 FY10 FY09_11 FY14 120 11 53 5732676 2016 1 3 1653 
4 1 FY10 FY09_11 FY14 120 11 53 5732676 2016 2 4 1539 
3 1 FY10 FY09_11 FY14 120 ck 53 5732676 2016 3 4 2560 
4 1 FY10 FY09_11 FY14 120 11 53 5732676 2016 4 2 1178 
5 1 FY10 Fy09_11 Fy14 120 1" 53 5732676 2016 5 2 897 
6 1 FY10 FY09_11 Fy14 120 ” 53 5732676 2016 6 2 413 
i 1 FY10 FYO09_11 FY14 120 1 53 5732676 2017 1 3 1558 
8 1 FY10 FY09_11 FY14 120 a 53 5732676 2017 2 4 1492 
9 1 FY10 FY09_11 Fy14 120 11 53 5732676 2017 3 4 2480 
10 1 FY10 FY09_11 FY14 120 1 53 5732676 2017 4 2 1214 
2 Initial Assessment of Experimental Results 


Since data farming generates a large amount of data, it is worth first 
exploring the output data graphically to get an initial understanding of OSAM’s 
modeling behavior. This section assesses OSAM'’s results by visualizing the total 
inventory composition and measuring the effects of the explanatory input factors 


on projected inventory strength for all replications of the experimental design. 


We start by visualizing the total projected operating inventory strength 
profile across graders for the projected period. Box plots of projected SWO 
inventory strength for grades O1—O6 by FY across all design points are shown in 
Figure 13. The boxplots reveal how variation in inventory strength changes over 
the projected FY. A smoother is added to emphasize the mean trend in inventory 
strength by FY. For grades O1 and O2, variation increases in the first few FYs 
before declining and leveling off beyond 2019 and 2021, respectively. Grades 
04, O5, and O6 appear to have a gradual increase in variation over the projected 
years. Grade O3 depicts the most variation, which also increases with FY. The 
large variation in this grade is partly due to the range of input values 
corresponding to the EPOCR policy. Grades O1, O02, and O5 appear to have a 
slightly negative trend while grade O3 has a more negative trend across all 
projected FYs. 


50 


Grade & Strength 
1 3 o 5 6 a 
3500 & Smooth 


3250 
3000 


2750 


2250 


2000 
1750 
1500 


Strength 


1250 


KK K—KENNNN KR KEKE RNANN KF RKP KERNAN NA KP RP RP KET A NN KE RP KP KERNAN AN RKP RP KE RANA 
NUN UNNNDNNNDNNANNNNNNNNNNANANANANAANATDNNNNANNNN NNN AN NNN 


Figure 13. Box Plots of SWO Projected Inventory Strength by Grade 
and FY for Experimental Design 


Additionally, of particular importance to N100 strength planners, is how 
OSAM’s different loss rate categories affect long-term inventory strength 
projections. Box plots again provide a manner in which to compare and contrast 
the variability in inventory strength results due to each loss rate category. Box 
plots in Figure 14 depict the variability in mean total inventory strength for 
FY2023 across all design points. The FY2010 loss rate appears to produce the 
lowest mean in total FY2023 projected inventory strength while FY2006 produces 
the highest mean. The variation in FY2023 projected total inventory strength is 
lowest with the FY2007 loss rate and highest with FY2015 loss rates. Appendix A 
contains inventory strength distribution plots and summary statistics 
corresponding to these plots. 
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Figure 14. Box Plot of SWO Total Mean Inventory Strength for FY 2023 
by Loss Rates for Experimental Design 


We further need to determine whether the differences in the mean total 
inventory strength results due to the different loss rate categories are caused by 
the effects of the loss categories rather than random effects in the modeling 
process. We cannot deduce such information from the box plots and a more 


robust statistical test is needed. The hypothesis tested is: 


° Ho : There is no difference in the mean of total inventory strength 
results in FY2023 due to OSAM’s loss rate categories, that is, 
U1=p2=.....- Ue, where the six populations refer to the six 
different OSAM loss rate categories or levels chosen for the study. 


© Hy: There is a difference in the means, that is, at least two means 


are different. 
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We use One-way Analysis of Variance (ANOVA) to test the hypothesis 
stated above. ANOVA tests differences in means between groups by partitioning 
the overall variance in the response into that due to each of the measured groups 
and the errors (Faraway, 2005). First, we test the assumptions of ANOVA to our 
data specifically that the mean inventory strength results across the loss rates 
categories are normally distributed and that the variance in the results is equal 
across the loss rate categories. We see in Figure 14 that these conditions are 
generally met, that is: (1) all the box plots depict a similar spread, suggesting 
equal variance among the loss rate category results; and (2) all the box plots 
appear symmetrical with centered interquartile ranges, suggesting normally 
distributed inventory strength results. An excerpt from the results of the ANOVA 
test is depicted in Figure 15. The y-axis is the mean total inventory strength 
results across all design points. The vertical span of the diamond shapes 
represents a 95% confidence interval around the mean of each loss rate 
category. The mean is represented by the center vertical line in each diamond. 
The size of the intersection of the circles measures the statistical difference in the 
mean total inventory strength results among each paired loss rate category, 
where the larger the intersection, the smaller the difference in means. Visual 
inspection of the ANOVA test reveals significant differences among the means. 
Statistical results shown in Figure 16 reveal a significant F-statistic, p-value < 
0.001, which supports the visual conclusion that there are significant differences 
in the inventory strength results produced by the loss rates. 
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Figure 15. One-way Analysis of Mean Variance of FY2023 Total 
Inventory Strength by Loss Rate 


Analysis of Variance 

Sum of 
Source DF Squares Mean Square F Ratio Prob > F 
LOSS_RATE 5 25488285 5097657 34.2947 = <.0001* 
Error 3162 470007562 148642 
C. Total 3167 495495847 


Figure 16. ANOVA Statistical Test for Loss Rates 


Similar ANOVA hypotheses are tested for transfer plans and accession 
plans to determine the variables’ respective effects on projected inventory 
strength. One interesting observation from Figure 17 is that having no transfers 
produces a significantly higher projected inventory strength value than when 
using a transfer plan. One-way ANOVA results by accession plans are shown in 
Figure 18. Projected inventory strength values for FY 2015 and 2014 accession 
plans are statistically indistinguishable and have the lowest mean. FY2012 
accession plan has the lowest projected inventory strength value, which is also 


statistically different from the values from the rest of the accession plans. 


54 


~* One-way ANOVA by Transfer Plan 
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Figure 17. One-Way Analysis of Mean-Variance of FY2023 Total 
Inventory Strength by Transfer Plan 
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Figure 18. One-Way Analysis of Mean-Variance of FY2023 Total 
Inventory Strength by Accession Plan 


a0 


3. EPOCR Policy 


We continue our initial analysis by determining the effects of implementing 
an EPOCR policy to operating inventory strength projections. For this, we 
summarize the mean inventory strength results across the design space by 
grade and FY and compare the results to Officer Programmed Authorizations 
(OPA). A composition of SWO inventory strength by grade and FY with Projected 
inventory strength compared against OPA is shown in Figure 19. We observe 
over-execution persisting in grade O3 in FY2016-2019, and gradually decreasing 
to healthy levels by FY 2022. FY2022 inventory strength summary statistics for 
each pay grade is shown in Table 7. 
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Figure 19. Projected SWO Inventory Strength with EPOCR Policy 
Against Officer Programmed Authorization by Grade and FY 
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Table 7. Summary Statistics of Mean SWO FY 2022 Projected Inventory 
Inventory Strength Composition Using ERPOCR Policy 


Grade Strength a Min Max eles ae OPA | Deviation 
1 1628 110.06 1427 1834 1540 1719 1988 -360 
2 1402 107.64 1202 1648 1310 1500 1441 
3 1840 358.24 1028 3036 1574 2061 1624 
4 1269 93.72 1031 1551 1205 1322 1101 
5 802 44.23 709 900 759 832 661 
6 435 24.81 386 484 408 457 238 
Total 7375 738.70 7053 322 


Figures 20 and 21 show projected inventory strength profiles for the 
lowest (FY2012) and highest (FY2015) accession plan, respectively. An 
interesting thing to note is that there is minimal difference between the low and 
high accession plan inventory projection profiles for grades O2 and O83, 
confirming our conclusions from the one-way ANOVA results for the accession 
plans. 
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Figure 20. Projected SWO Inventory Strength with EPOCR 
Using a Low Accession Plan 
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Figure 21. Projected SWO Inventory Strength with EPOCR 
Using a High Accession Plan 


4. Regression Tree Analysis 


Regression trees offer a result structure that is more intuitive and are less 
computationally tasking than linear or polynomial regression. Additionally, they 
are powerful at identifying influential factors affecting an outcome variable, as 
well as finding interactions among variables (Faraway, 2005). The general 
regression tree methodology also allows analysis for a combination of discrete 
and categorical predictor variables as used in this study. 


We use regression tree analysis to determine the most influential factors 
of total SWO inventory strength for grades O1 to O6 in FY2022. The results of 
the regression tree model for total inventory strength are shown in Figure 22. The 
first node, the parent leaf, or root leaf, shows the mean and standard deviation of 
the total inventory strength in the entire dataset, 7,375 and 365, respectively, and 


the total count of all observations, 3,168. The data is recursively partitioned into 
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regions corresponding to the distinct paths from the parent leaf to the leaves with 
no further splits, the terminal leaves. The intermediate cells between the parent 
and terminal leaves are child leaves. A predictive model is fitted for each leaf and 
a corresponding estimate of the mean total inventory strength is determined for 
that leaf. The title cell shows the number of splits and the R-square value, which 
describes how well the tree explains the variance in mean total projected 
inventory strength results. The most influential factors in determining the 
projected mean total inventory strength in order of importance in the tree are 
transfer plans, EPOCR policy with 4 YCS, and EPOCR with 5 YCS. The R- 
Square value of 0.733 suggests that the tree model explains about 73% of the 
variation in total end strength results. An expected observation, as revealed in 
the first split of the tree, is that having no transfer plan significantly increases the 
total mean inventory strength. Each distinct path from the terminal to the parent 
leaf also represents a sequence of interacting terms. Using a transfer plan with 
an EPOCR policy that refers 64 or more officers with four YCS, and 53 or more 
officers with 5 YCS will results in the least mean total inventory strength. On the 
contrary, having no transfers and referring fewer than 64 officers with four YCS 
results in the largest projected inventory strength value. 
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rae 


RSquare RMSE N of Splits AlCc 
0.733 18861011 3168 5 42203.1 


* All Rows 

Count 3168 LogWorth Difference 
Mean 7375.3475 575,14531 549.573 
StdDev 365.38147 


» TRANS PLAN(FY14, FY15, FY09_ 10) ¥ TRANS PLAN(No Transfer) 
2376 LogWorth Difference 792 LogWorth Difference 
7237.9542 307.06766 312.267 787.5275 125.18786 315.448 
281.4262 StdDev 264.40298 


¥ EPOCR (YCS 4)>=64 ¥ EPOCR (YCS 4)<64 
Count 1152 LogWorth Difference Count 1224 LogWorth Difference 

Mean 7077.0896 260.7802 323.886 Mean 7389.3562 211.74072 298.016 7625.024 7940.4721 
StdDev 236.18316 StdDev 232.3544 StdDev 213.66955 StdDev 211.04533 


> Candidates > Candidates 


* EPOCR (YCS 5)>=53]| * EPOCR (YCS 5)<53| | * EPOCR (YCS 5)>=90]|\* EPOCR (YCS 5)<90 
Count 720 Count 432 Count 432 Count 792 
Mean 6955.6325 Mean = 7279.5181 Mean 7196.5222 Mean 7494.5384 
StdDev 169.1605 StdDev 188.45911 StdDev 189.84767 StdDev 180.13844 


> Candidates > Candidates > Candidates > Candidates 





Figure 22. Regression Tree for SWO Total Inventory Strength 
in FY2022. 


D. ANALYSIS OF OPERATING STRENGTH DEVIATION 


Of importance to this research, and of interest to N100, is determining the 
best manpower policies that will minimize the deviation of projected SWO 
inventory strength from planned authorizations at a given future period. We will 
specifically focus on the total FY2022 projected inventory strength for grades O3 
to O6. We have already determined that there is significant variation in mean 
total inventory strength across the different manpower policies used in this study. 
Furthermore, we have identified the most influential factors that affect projected 
total inventory strength. This section uses tree analysis to find the most influential 
factors that determine FY2022 total OSD for grades O3 to O6. Concepts of 
robust design and meta-modeling are used to determine manpower policy 


settings that minimize projected total OSD in these grades. 
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1. Regression Tree Total Operating Strength Deviation 


We use regression tree analysis to identify influential factors determining 
total operating strength deviation (OSD) over the loss rates for grades O3 to 
grade O6. To set up our dataset for this analysis, we subset the parent dataset to 
grades O3 to O06 and FY2022 and introduce a column in the resulting dataset 
containing the OSD at each design point. We then sum OSD over the grades by 
all the input factors and random seeds. The resulting format of the data with a 
column for the sum of OSD represented as sum (Deviation) is shown in Table 8. 
We use JMP to build a regression tree model on the mean total OSD. The 
resulting regression tree is shown in Figure 23. The variables in the regression 
tree model at six splits explain about 63% of the variation in OSD. Transfer plan, 
EPOCR 5 YCS, and, EPOCR 4 YCS are most significant in determining the 
amount of OSD. 


Table 8. Data Format with Inventory Strength Deviation Column 


° EPOCR | EPOCR | EPOCR N Sum 
LOSS RATE TRANS PLAN ACC PLAN (YCS4) (YCS5) (YCS6)| seed | Rows (Deviation) 
1 FY06 FY09_10 FY12 0 109 68 14143576 4 482 
2 FYO6 FY09_10 FY12 0 109 68 35989036 4 531 
3 FYO6 FY09_10 FY12 0 109 68 46807864 4 528 
4 FY06 FY09_10 FY12 0 109 68 56621581 4 518 
5 FY06 FY09_10 FY12 0 109 68 80735315 4 518 
6 FYO6 FY09_10 FY12 4 8 83 255496 4 864 
7 FYO6 FY09_10 FY12 4 8 83 29743319 4 850 
8 FYO6 FY09_10 FY12 4 8 83 31222864 4 821 
9 FYO6 FY09_10 FY12 4 8 83 68509862 4 840 
10 FYO6 FY09_10 FY12 4 8 83 73352979 4 838 
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Split || Prune Number 


"Square RMSE —N of Splits AlCe 
0,633 24353609 3168 6 438244 


72180802 426.7747 «552.113 
Std Dev 40204627 


* TRANS PLAN(FY14, FY15, FYO9 10) 
Count 2376 LogWorth Difference 


Mean 583.7798 22830239 326.626 
StdDev 32353628 


TEPOCR (YCS 5)>=53 TEPOCR (CS 5)<53 

Count 1368 LogWorth Difference Count 1008 LogWorth Difference 

Mean 44521126 16816655 333.164 Mean 771.8371 130.6874 319.228 
StdDev 261.17612 


* EPOCR (YCS 4)>=49 ; * LOSS RATE(FYO7, FY6))) * LOSS RATE(FY14, FY10, FY13, 
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Mean  322.46667 109,03303 1 5.63 Mean  559,01845 Count 672 

Std Dev 228,03063 274, Std Dev 199.73489 Mean 87824643 


> Candidates > Candidates a Dev 220.0762 
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* TRANS PLAN(No Transfer) 
Count 792 
Mean 1135,8927 
Std Dev 32245237 


> Candidates 


Std Dev 149.62837 StdDey 15249284 
» Candidates » Candidates 
Figure 23. Regression Tree of Mean total Operating Strength Deviation 
for grades O3 to O6 


2. Robust Design and Construct 


Robust design is a powerful system optimization technique that takes the 


approach of finding ways to determine, with consistency, accurate and precise 


response values by controlling for uncertainty in the system. Both the mean and 


variation of the response are considered, where a system is optimized by 


choosing the closest mean to a target value with the smallest variance (Sanchez, 


2000). The idea is to select alternatives that best approximate 


the desired 


threshold value for their mean, as well as produce a reasonably small variance 
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around that threshold. Variability in robust analysis, then, represents a risk or 
cost in the system modeled. 


The approach in robust analysis is to separate the explanatory factors of a 
response variable into controllable and uncontrollable factors and then use a loss 
function to find the best configuration of the explanatory factors that produces the 
best results with minimal variability across uncontrollable factors. The response 
variable of interest is the operating strength deviation and the controllable 
variables are represented by the manpower plans: accession plan, the transfer 
plan, and the three EPOCR variables. Loss rate categories represent 
uncontrollable factors corresponding to the uncertainty in the real world. Loss or 
risk is a function of the variance and mean of the operating strength deviation, 
and is given by the formula 


E(Loss) = variance(deviation) + (mean(deviation))* (4.1) 


where E(Loss) is the long run cost or risk of operating strength deviation, and 
where zero is the best possible value of the deviation. The loss function 
penalizes for both a higher variance and for the mean straying away from its best 
possible value. In other words, if both values are low, the manpower 
configuration is robust and should perform consistently well in minimizing 
operating strength deviation. The second term is squared to level the units for 
both terms to units-squared. 


To set up our data for robust analysis, we further process Table 8 data by 
averaging the mean and variance of the summed deviation over the loss rates. 
The resulting dataset is depicted in Table 9. We then proceed to build a meta- 
model on operating strength deviation. 
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Table 9. Data Format of Robust Design Dataset with Loss Column. 


v EPOCR EPOCR EPOCR Mean Variance 
TRANS PLAN ACC PLAN (YCS_4) | (YCS_5) | (YCS_6) N Rows (Sum(Deviation)) (Sum(Deviation)) 

1 FY09_10 FY12 0 109 68 30 655.567 30437.426 
2 | FY09_10 FY12 4 8 83 30 1011.200 36346.579 
3 FY09_10 FY12 8 116 64 30 615.967 30008.171 
4 FY09_10 FY12 a4 0 105 30 925.000 34487.655 
5 FY09_10 FY12 15 68 11 30 1032.733 34342.961 
6 | FY09_10 FY12 19 49 is) 30 816.300 32853.734 
7 FY09_10 FY12 23 45 49 30 934.000 34426.897 
8 | FY09_10 FY12 26 34 41 30 1006.100 34183.197 
9 FY09_10 FY12 30 41 86 30 760.067 32377.789 

10 | FY09_10 FY12 34 94 30 30 766.333 31219.126 


3. Meta-Models for Mean and Variance of Operating Strength 
Deviation 


We extend our analysis to build a meta-model that determines robust 
parameters for correcting FY2022 total operating strength deviation for grades 
03-06. Meta-models act as proxies to the entire simulation model by offering a 
more computationally efficient way of computing response estimates from the 
simulation model (Law, 2000). We use meta-modeling to understand the 
relationship between the controllable input factors’ settings for the accession 
plans, transfer plan, and the three EPOCR variables with the response variables 
represented by the mean and variance of OSD. The meta-model quantifies the 
effects of each of the controllable variables on both the mean and variance of 
operating strength deviation. 


Using JMP, we simultaneously fit meta-models on the mean and variance 
of operating strength deviation using stepwise regression and with minimum 
Bayesian information criterion (BIC) selected as the stopping condition for fitting 
the best model. We screen out less significant factors from the resulting models 
and use the remaining factors to fit a model using all two-way interactions and 
second-degree polynomials terms. We further filter out insignificant factors from 
the resulting models, leaving only statistically significant factors and interactions. 
We use data splitting to avoid over-fitting and to ensure that the models have 
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significant prediction power. For this, the dataset used to build the model, Table 
9, is randomly partitioned into a training set that is 75% of the data, and is used 
to build the models. The rest of the data is set aside as a test set and is used to 


validate the predictive power of the model. 


A screen shot of summary statistics and diagnostic plots for the resulting 
meta-model built with the training set is depicted in Figure 24. The summary 
statistics indicate good predictive capability. All five input factors are statistically 
significant with t-statistic p-values < 0.01. Additionally, the model has a relatively 
high R-Square value of 0.999. The residuals are also sufficiently normal and 
exhibit constant variance, as evidenced by the Normal Quantile and Residual by 
Predicted plots, suggesting that the model has a good fit on the data. A bivariate 
fit of predicted values from the training and test set is shown in Figure 25. The 


linear diagonal fit indicates that the model has good predictive performance. 
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4 Summary of Fit 
RSquare 
RSquare Adj 
Root Mean Square Error 
Mean of Response 


Observations (or Sum Wgts) 


4 Effect Tests 


Source Nparm DF 


TRANS_PLAN 3 
ACC_PLAN 3 
SWO_YCS 4 1 
SWO_YCS_5 1 
SWO_YCS 6 1 


4 Residual by Predicted Plot 


8 
6 © 


4 


Mean (Sum(Deviation)) Residual 
c: 


3 
3 
1 
1 
1 





0.999951 
0.99995 
2.977441 
724.2183 
394 


Sum of 
Squares F Ratio Prob > F 
25392517 1274111 <.0001" 
438562 2200557 <001* = Residual Mean (Sum(Deviation)) 
6636151 9989388 <.0001" 
9553868 1438142 <.0001" 
9559031 1438919 <,0001" 


0.002 
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Figure 24. Diagnostic Plots for Mean Total End Strength Deviation 
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Normal Quantile Plot 
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Figure 25. Actual versus Predicted Plot for Test Set Using 
Mean Total Inventory Strength Meta-model 


4. Prediction Profiler 


JMP’s prediction profiler offers an interactive way of exploring a model’s 
response surface by varying the input factors. The profiler displays the predicted 
mean and variance of operating strength deviation as one input variable is 
changed while holding the others constant. We can use the prediction profiler to 
find the robust settings that minimize both the mean and variance of the OSD 
variable. A prediction profiler with function curves for the mean and variance of 
deviation is shown in Figure 26. The transfer plan and accession plan are set to 
FY2015 and EPOCR parameters are set to zero. This corresponds to the status 
quo where no EPOCR policy is implemented. Shifting the input parameters on 
the x-axis along the deviation curves results in a marginal change in deviation 
proportional to the slope of the curve. Thus, a curve with a slope of zero will have 
no effect on the response. The response at these values indicates a mean OSD 
of 1261 with a standard deviation of 189, suggesting that total projected 
operating inventory strength in grade O3 to O6 will over-execute OPA by an 
estimated 1261 officers in FY2022 where an EPOCR plan is not implemented. 
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This again assumes that that Navy implements the respective manpower policies 


each year over the projected period. Shifting the transfer plan to no transfers 
increases the mean OSD, as does shifting the EPOCR value setting to the right. 
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Figure 26. Prediction Profiler of Mean-Deviation Meta-Model 


Figure 27 depicts a prediction profiler with transfer and accession plans 
set at their respective values from the robust analysis. The EPOCR parameters 
are arbitrarily set to a combination such that OSD is reduced to a modest level of 


300 with a standard deviation of 141. This suggests that the total inventory for 


grades O3 to O6 will over-execute OPA by a mean estimate of 300 officers in 


FY2022 assuming this policy prevails over the projected period. 
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Figure 27. Prediction Profiler of Mean-Deviation Meta-Model With 
Robust Settings for Transfer and Accession Plans. 


Figure 28 depicts a prediction profiler with robust settings for transfer and 
accession plans with EPOCR parameters arbitrarily set to a combination that 
reduces OSD to a mean estimated value of just two officers in FY2022. The 
standard deviation for this setting is 123 officers, suggesting an enhanced risk of 
under-executing SWO inventory. 
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Figure 28. Prediction Profiler with Robust Settings Converging 
Operating Strength Deviation to two Officers 
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V. CONCLUSION AND RECOMMENDATIONS 


This thesis extends the works of Sibley (2012), Borozny (2015), and 
DeHollan (2015) to enhance OSAM’s utility as an inventory projection model 
using data farming and design of experiments. The insights gained from this 
study go beyond the scope of the specific questions guiding this research. To 
illustrate OSAM’s utility, this thesis, first, analyzes the effects of current 
manpower policy on the future composition of URL inventory. Second, this thesis 
analyzes the effects of historical accession plans and the proposed EPOCR 
policy on surface warfare officer inventory. Finally, the study identifies the risk 
inherent in the EPOCR policy by measuring projected inventory deviation from 
planned authorizations and identifies policies that mitigate identified risk. The 


section that follows answers the specific research questions posed for this study 


A. ANSWERS TO RESEARCH QUESTIONS 


1. What Is the Expected Short- and Long-Term Trend in URL 
Operating Inventory Assuming Current Policy and Prevailing 
Assumptions? 


OSAM models inventory projections dynamically over annual time-steps, 
and as such, provides the ability to track URL inventory over the entire projected 
period. The base-case projects 2016 URL inventory over a seven-year period 
with a key assumption that the current policy, FY2015 accessions, transfer plans, 
and loss rates prevail over the projected period. We find that total operating 
inventory strength falls by an average of 8% over the seven-year period from 
24,815 officers at the beginning of 2016, to a mean and standard deviation of 
22,890 and 13 officers at the end of 2023. The mean Inventory strength results in 
the base case have extremely small standard deviation values, suggesting little 


value in using OSAM as a stochastic model. 
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2. What Are the Long-Term Risks in Inventory for the SWO 
Community Associated with Current Policy? 


We measure risk as operating strength deviation, the magnitude of 
expected deviation of operating inventory from planned authorizations (OPA). 
The base-case results show over 40% over-execution in grade O38 that is 
expected to persist over the entire projected period. We should remind the reader 
that the base-case assumes current manpower policy and loss rates prevailing 
over the projected period. In addition, the current accession plan is not enough to 
sustain an increase in programmed authorizations for grade O1 planned for 
FY2017 and beyond. Interestingly, the base case shows healthy inventory 
strength for grades O04, O5, and O6. 


3. What Is the Impact of the Enhanced Probationary Officer 
Continuation and Re-designation Board on SWO Inventory? 


The EPOCR results assume averages over all practical manpower 
policies as used in this study. Additionally, the results rest on the assumption that 
the policies prevail over the projected period. We find that implementing the 
policy significantly reduces grade O83 over-execution by FY2022, and levels the 
average operating strength in all grades to match OPA. Total operating inventory 
strength is reduced by 12% over the seven-year period, from 8,365 officers at the 
beginning of FY2016 to a mean and standard deviation of 7,375 and 739 officers 
at the end of FY2023. Total SWO over-execution is reduced to a modest 4% over 
FY2022 OPA from 34% in FY2015. The extremely high standard deviation on 
FY2022 total SWO inventory suggests a high risk of under-executing. Whereas 
the EPOCR policy will generally have a correcting impact on operating strength 
deviation, caution should be taken on how to implement it to avoid under- 
execution. Robust analysis of the policy can identify how to best implement 
EPOCR. 


Focusing on the total inventory of grades O3 to O6, we use robust 
analysis and meta-modeling to determine the risks in implementing EPOCR, and 
find the best manpower policies that will correct operating strength deviation in 
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these grades. Our initial finding suggests that using a transfer plan and 
implementing an EPOCR policy for surface warfare officers with 4 and 5 YCS 
had the most significant effect on total OSD. We find that a low accession plan, 
that of FY2012, and a high transfer plan, that of FY2014, are the most robust 
non-EPOCR policies to correct OSD. All accession plans used in the study had 
relatively low impact on OSD, suggesting that an economically efficient plan, a 
low accession plan, holds the same risk as a high accession plan. Additionally, 
the FY2014 transfer plan produced the lowest OSD. Prediction profilers on meta- 
models of OSD provide evidence that EPOCR can effectively correct operating 
end-strength to OPA. 


The study shows that it is possible to meet future SWO manpower 
requirements with a conservative accession plan coupled with an aggressive 
transfer plan referring officers to other communities. The Navy should additionally 
supplement its current force-shaping tools with EPOCR to manage officer over- 
execution. When used correctly, EPOCR has a potential to minimize over- 
execution in the SWO community with minimal and measurable risk of under- 
executing. The number of officers within each YCS group refereed to EROCR 
should be selected in such a manner to account for other constraints, 
assumptions, and interests external to the scope of this research. 


B. RECOMMENDATIONS FOR FUTURE STUDIES 


This study demonstrates the utility of OSAM as an officer projection model 
by applying efficient methods of extracting information from the model and 
providing a quantitative framework to analyze the model’s results. There exists 


potential for research that builds on this study or improves on OSAM's utility. 


1. Continuous Input Levels 


Future work on OSAM can effectively enrich the factor space in the 
experimental design by varying beginning of year inventory, loss rates, transfer 
plans, and accession plans as continuous rather than categorical variables. The 


current version of OSAM, and as used by this study, represents these input 
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variables as a set of pre-established tables corresponding to historical 
realizations of these plans, given by FY, designator, and paygrade. Since OSAM 
retrieves the inputs as numerical sets, loss rates, transfer plans, and accessions 
are modeled as categorical variables in the experimental design construct, where 
the factors in the experimental design correspond to the manpower plans, and 
the factor levels to a numeric constant for each designator and grade. The factor 
levels do not cover a realistic range of possibilities for the manpower plans. 
Additionally, we cannot prescribe numeric values for the manpower plans in our 
results. However, using OSAM’s database structure, we can use an adjustment 
factor to effectively update input values for the plan parameters in the 
experimental design through SQL statements. For example, 


UPDATE AccPlan_FY16 SET Accessions=Accessions*1.5 WHERE 
‘Community’ =SWO TRAINEE 


will increase FY2016 SWO accessions by a factor of 1.5. BOY inventory, loss 
rates, and transfer plan inputs can be adjusted in a similar way. We can then 
vary the adjustment factors in the experimental design to construct a realistic 
range of numeric values that cover the manpower plans. An experimental design 
varying the manpower plans as discrete factors would give a more precise 
assessment as to how the numeric levels of the manpower plans affect projected 
inventory strength. 


2. Fiscal Year Fixed-Effects 


This thesis also uses linear projections of OSAM inventory over the 
projected window. The manpower and force-shaping plans (EPOCR) are 
modeled with the assumptions that the specific input parameters for each 
scenario are implemented linearly and annually over the projected window. 
Further research should consider projected year fixed-effects for each of the 
manpower and force-shaping plans. The advantage of such a construct is that 
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the research can identify an optimal manpower policy for each projected fiscal 
year. We will add to this note that adding fixed-year effects will compound the 
number of factors needed by a factor proportional to the number of projected 
years. However, the computational resources needed to account for additional 
factors are minimized by using an appropriate experimental design. Additionally, 
since OSAM has very little variation, replication adds little value to an 
experiment. Construction of any designed experiment for future work should be 


weighted in favor of more design points rather than number of replications. 
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APPENDIX A. SWO PROJECTED INVENTORY BY GRADE AND 
FY ACROSS DESIGN POINTS-FY2023. 
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APPENDIX B. DISTRIBUTION OF PROJECTED END STRENGTH 
BY LOSS RATE CATEGORY-FY2023. 


4 = Distributions LOSS RATE=FY10 
4 ~ Sum(Mean(Strength)) 





4 Summary Statistics 


Std Dev 390.50635 
Std Err Mean 16.994608 
Upper 95% Mean 7183.5336 
Lower 95% Mean 7116.7626 
N 528 


6200 6500 6800 7100 7400 7700 8000 


4 ~ Distributions LOSS RATE=FY13 


4 ~/Sum(Mean(Strength)) 
4 Summary Statistics 


}—__1__$__ J |_ Mean 72566818 


Std Dev 406.42045 
Std Err Mean 17.68718 
Upper 95% Mean 7291.4279 
Lower 95% Mean 7221.9358 
N 528 


6300 6600 6900 7200 7500 7800 8100 8400 


4 ~ Distributions LOSS_RATE=FY14 
4 ~ Sum(Mean(Strength)) 





4 Summary Statistics 

+——__{___¢ } +4 Mean 7347.9519 
Std Dev 397.56849 
Std Err Mean 17.301948 
Upper 95% Mean 7381.9411 
Lower 95% Mean 7313.9626 
N 528 





a 
6500 6800 7100 7400 7700 8000 8300 


79 


4 ~ Distributions LOSS RATE=FY15 
4 ~ Sum(Mean(Strength)) 
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4 ~/Summary Statistics 
Std Dev 417.71115 
Std Err Mean 18.178544 
Upper 95% Mean 7353.503 
Lower 95% Mean 7282.0804 
N 528 


4 ~)/Summary Statistics 


Mean 7439.6864 
Std Dev 343.03445 
Std Err Mean 14.928658 


Upper 95% Mean 7469.0133 
Lower 95% Mean 7410.3594 
N 528 


4 ~ Summary Statistics 
Mean 7255.4235 
Std Dev 352.08836 
Std Err Mean 15.322679 
Upper 95% Mean 7285.5245 
Lower 95% Mean 7225.3225 
N 528 
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